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^The  present  study  was  designed  (l)'to  assess  the  effects  of  various 
follow-up  techniques  on  the  return  rate  and  response  bias  of  demographic 
and  attitudlnal  measurements,  and  (2)  to  explore  the  possibility  that 
reaction  to  extensive  folloie-ups  biases  the  responses  from  initially 
uncooperative  selectees. 

Approximately  1500  Navy  and  Marine  Corps  personnel  from  San  Diego,  were 
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mailed  tlie  Dol)  Family  Housing  Preference  Survey  and  were  assigned  to  10 
follow-up  groups.  Eight  of  the  groups  formed  a completely  crossed 
2x2x2  design,  defining  the  follow-up  by  type  of  mall,  type  of  material, 
and  number  of  follow-ups.  Two  additional  groups  were  formed,  one  receiving 
no  follow-up  and  the  other  receiving  extensive  follow-up.  Both  return 
rate  and  response  bias  on  12  attitudinal  and  demographic  variables  were 
studied . 

The  results  in  this  study  indicate  that  some  type  of  follow-up  pro- 
cedure results  in  greater  return  rates  than  no  follow-up.  Response  bias 
was  found  in  some  variables.  ~ 
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FOREWORD 


This  study  was  sponsored  by  the  Naval  Facilities  Engineering  Command 
(NAVFACENGCOM) . It  was  conducted  in  conjunction  with  the  Department 
of  Defense  Family  Housing  Preference  Survey  of  1975. 

All  phases  of  the  project  were  directed  by  E.  P.  Somer  to  whom 
a special  word  of  appreciation  is  due  for  his  valuable  suggestions  and 
guidance  throughout  the  project. 


J.  J.  CLARKIN 
Commanding  Officer 
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SUMMARY 


Problem 

A widely  held  view  In  survey  research  Is  that  response  bias  significantly 
affects  population  parameter  estimates.  Numerous  studies  have  documented 
effects  on  demographic  variables,  but  few  have  researched  the  effects  on 
attltudlnal  measures. 

Purpose 

The  present  study  was  designed  to  assess  the  effects  of  various  follow- 
up techniques  on  return  rate  and  response  bias  in  mail  surveys  involving 
demographic  and  attltudlnal  measurements,  and  to  explore  the  possibility 
that  negative  reaction  to  extensive  follow-ups  biases  the  responses  from 
Initially  uncooperative  selectees. 

Approach 

Approximately  1500  Navy  and  Marine  Corps  personnel  from  San  Diego  who 
received  the  DoD  Family  Housing  Preference  Survey  were  assigned  to  10  follow- 
up groups.  Eight  of  the  groups  formed  a completely  crossed  2x2x2 
design,  defining  the  follow-up  by  type  of  mall  (special  delivery  or  regular 
mall),  type  of  material  (letter  only  or  letter  and  additional  questionnaire 
with  return  envelope),  and  number  of  follow-ups  (one  or  two).  Two  additional 
groups  were  formed,  one  receiving  no  follow-up  and  the  other  receiving 
extensive  follow-up. 

Both  return  rate  and  response  bias  on  12  attltudlnal  and  demographic 
variables  were  studied  using  a three-way  analysis  of  variance  to  compare 
the  eight  treatment  groups,  and  two  linear  contrasts  to  compare  them  to 
the  no  follow-up  and  extensive  follow-up  groups. 

Findings 

Using  a significance  level  of  .01,  the  eight  treatment  groups  were  not 
found  to  differ  on  return  rate,  but  did  differ  significantly  from  both  the 
no  follow-up  group  and  the  extensive  follow-up  group. 

Regarding  response  bias,  two  of  the  attltudlnal  variables  indicated 
differences  in  means  between  the  no  follow-up  groups  and  the  eight  treat- 
ment groups,  but  no  differences  were  found  between  the  eight  treatment 
groups  and  the  extensive  follow-up  group.  The  analyses  of  variance  among 
the  eight  treatment  groups  indicated  inconsistent  effects,  with  four 
significant  main  effects  and  six  significant  Interactions. 

Late  respondents  were  expected  to  be  more  neutral  on  satisfaction  and 
policy  Items  than  the  early  respondents,  but  this  expectation  was  not  con- 
firmed . 
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Conclusions 


This  study's  results  support  those  of  earlier  studies,  which  indicated 
that  employing  some  type  of  follow-up  elicits  greater  return  rates  than 
using  no  such  procedure.  With  the  exception  of  extensive  follow-up,  there 
was  no  evidence  that  one  follow-up  technique  is  more  effective  than  another. 

The  population  parameters  of  the  attitudes  investigated  may  be  more 
accurately  estimated  by  obtaining  a larger  sample  of  respondents  through 
follow-up  treatments.  However,  if  the  Issues  studied  are  not  critical, 
and  small  differences  in  any  one  of  the  variables  carry  low  or  no  risks, 
then  a follow-up  need  not  be  obtained. 

Recommendations 

Further  studies  of  respondents  should  be  carried  out  to  explore  the  con- 
sistency of  differences  between  various  treatment  groups  when  the  issues 
and  population  differ  from  those  in  this  study.  Investigators  employing 
mail  surveys  should  attempt  to  determine  whether  a bias  exists  through  the 
use  of  a follow-up  on  a subsample  of  nonrespondents.  In  addition,  a letter 
follow-up  is  recommended  over  a duplicate  package  of  materials  by  special 
delivery,  due  to  the  lower  cost  and  the  failure  to  produce  a consistent 
bias. 
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INTRODUCTION 


Problem 


A widely  held  view  in  survey  research  is  Chat  response  bias  significantly 
affects  population  parameter  estimates.  There  are  numerous  studies  docu- 
menting demographic  differences  between  early  respondents,  late  respondents, 
and  nonrespondents.  However,  such  demographic  differences  do  not  necessarily 
indicate  that  the  responses  to  attitudinal  measures  are  also  affected  by  a 
response  bias.  A need  therefore  exists  for  empirical  testing  of  this  assump- 
tion using  attitudinal  measures  typical  of  survey  research. 

Survey  follow-ups  appear  to  be  consistently  successful  in  increasing  the 
return  rate  (Bachrach  & Scoble,  1967;  Donald,  1960;  Eckland,  1965;  Goldstein  & 
Kroll,  1957;  Hinrichs,  1975;  Levine  & Gordon,  1958;  Scott,  1961;  Suchman  & 
McCandless,  1940).  Since  follow-ups  almost  always  increase  return  rates, 
it  is  worthwhile  to  know  what  kinds  of  and  how  many  follow-ups  most  effec- 
tively increase  return  rate  and/or  decrease  response  bias. 

Because  so  few  studies  have  employed  simultaneous  comparisons  of  several 
different  follow-up  techniques  in  relation  to  both  return  rate  and  response 
bias  (Friedman  & Goldstein,  1975;  Field,  1975;  Nevin  & Ford,  1976;  Panos  & 
Rice,  1967),  there  is  a need  for  further  investigation  in  these  areas. 

Purpose 

The  present  study,  which  was  designed  to  assess  the  relative  values 
of  several  methods  of  follow-up  in  a mail  survey,  concerns  three  factors 
of  follow-ups:  type  of  mail,  type  of  material,  and  number  of  follow-ups. 

The  specific  purposes  are: 

1.  To  compare  the  effects  of  various  follow-up  techniques  (including 
no  follow-up)  on  the  obtained  return  rate. 

2.  To  determine  whether  population  parameter  estimates  of  attitudes 
are  affected  by  the  use  of  various  follow-up  techniques  and  to  compare  these 
estimates  with  those  obtained  when  no  follow-up  is  used. 

3.  To  test  the  theory  that  individuals  react  negatively  to  increased 
urging  to  respond  to  a mail  questionnaire  survey  and  that  this  reaction 
biases  responses  to  attitudinal  items. 

Background 

Return  Rate 


"Type  of  mail"  has  been  covered  extensively  in  the  literature  on 
return  rates  for  the  original  mailing.  Few  studies,  however,  have  concerned 
themselves  with  the  type  of  mail  employed  in  the  follow-up.  Landy  and  Bates 
(1973)  found  neither  main  nor  interaction  effects  for  return  rate  in  a 
3x3x3  (type  of  postage  x degree  of  personalization  x nonmonetary  reward) 
design.  Special  delivery  follow-ups  are  generally  more  effective  than  first- 
class  mail  follow-ups  (Blumberg,  Fuller,  & Hare,  1974;  Champion  & Sear,  1969; 
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CuI  Inhorn  6 Gullahorn,  1 0 5 *1 ; Kephart  & Breasler,  .1958;  Phillips,  1951; 

Slocum,  Empey,  & Swanson,  1958),  and  hand-stamped  envelopes  are  generally 
more  effective  than  postage-permit  envelopes  (Champion  f«  Sear,  I960;  Gullahorn 
& Gullahorn,  1959;  Hammond,  1^59;  Kimball,  1961;  Robinson  & Agism,  1951; 
Watson,  1965). 

Very  few  studies  have  been  concerned  with  the  type  of  material  used 
in  the  follow-up.  A postcard  follow-up  was  shown  to  be  as  effective  as 
a letter  follow-up  (Sletto,  1940;  Watson,  1965).  Sheth  and  Roscoe  (1975), 
using  a 2 x 4 x 8 factorial  design  (questionnaire  length  x type  of  follow- 
up method  x geographical  region),  found  that  a telephone  reminder  was  most 
effective  and  a letter  alert  followed  by  a telephone  interview,  least  effec- 
tive. Only  one  study  (Etzel  & Walker,  1974)  investigated  the  relative 
advantages  of  including  a new  questionnaire  with  the  follow-up  letter, 
finding  that  the  response  rate  among  the  group  receiving  only  the  letter 
was  slightly  (but  not  significantly)  higher  than  that  for  the  group  re- 
ceiving an  additional  questionnaire. 

Thus,  only  a few  of  the  studies  cited  above  investigated  several 
follow-up  methods  simultaneously  or  compared  special  delivery  and  regular 
mail  follow-ups.  Only  one  compared  a letter-only  follow-up  to  a letter 
and  a new  questionnaire. 

Response  Bias 

Response  bias  has  been  studied  by  determining  whether  differences 
exist  between  respondents  and  nonrespondents  or  between  early  respondents 
and  late  resondents.  The  findings  can  be  grouped  into  three  variable 
categories;  demographic,  attitudinal,  and  other  (behavioral,  personality, 
and  aptitude). 

Demographic  Variables.  The  literature  relating  demographic  vari- 
ables to  survey  response  behavior  is  extensive  and  shows  differences  in 
income,  age,  education,  marital  status,  sex,  family  size,  and  ethnicity 
(Baur,  1947;  Bell,  1961;  Benson,  Booman,  & Clark,  1951;  Britton  & Britton, 
1951;  Clausen  & Ford,  1947;  Davis,  1964;  Donald,  I960;  Ellis,  Endo,  & 

Armor,  1970;  Ferber,  1956,  Franzen  & Lazarsfeld,  1945;  Gannon,  Nothern,  & 
Carrol,  1971;  Gaudet  & Wilson,  1940;  Hilgard  & Payne,  1944;  Hochstim  & 
Athanasopoulos,  1970;  Hood  6 Back,  1970;  Kirk  & Sereda,  1969;  Lowenstein, 
Colombotos,  & Elinson,  1962;  Eundberg  & Larsen,  1949;  Mayer  & Pratt,  1966; 
Ognibene,  1970,  1971;  Parten,  1950;  Robins,  1963;  Roeher,  1963;  Rosnow, 
Rosenthal,  McConochie,  & Arms,  1969;  Sobol,  1959;  Stonborough,  1942; 

Suchmnn  & McCnnd1os9,  1940;  Suchman,  1962;  Vincent,  1964;  Wallace,  1954). 

Attitude  Variables.  Several  studies  have  also  demonstrated  that 
respondents  tend  to  have  greater  interest  in  the  topic  or  the  sponsoring 
organization  than  nonrespondents  or  late  respondents  (Ball,  1930;  Bell, 

1961;  Donald,  1960;  Edgerton,  Britt,  6 Norman,  1947;  Franzen  f.  Lazarsfeld, 
1945;  Larson  & Catton,  1959;  Lundberg,  1942;  Newman,  1962;  Pace,  1939; 
Scheverian  & Blaine,  1966,  Shutt leworth,  1941;  Stanton,  1939). 
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Broedling  and  Fuller  (1976),  surveying  the  same  sample  four  times 
with  four  separate  surveys,  used  the  proportion  of  surveys  returned  as  an 
index  of  each  subject’s  tendency  to  respond  to  mail  surveys.  A number  of 
demographic  characteristics  were  related  to  response  tendency  but  no  rela- 
tionship was  found  between  response  tendency  and  the  attitudes  assessed 
by  each  item  individually. 

Other  Variables  (Behavioral,  Personality,  and  Aptitude).  Ognibene 
(1970)  found  respondents  to  differ  from  nonrespondents  on  leadership, 
gregariousness,  and  reading  habits.  Lubin,  Levitt,  and  Zuckerman  (1962) 
found  respondents  to  be  more  orderly  and  dependent,  but  less  aggressive, 
dominant,  autonomous,  and  intraceptive . Vincent  (1964)  found  respondents 
to  be  more  responsible,  tolerant,  and  intellectual.  Macek  and  Miles 
(1975)  found  respondents  to  have  a higher  mean  IQ  than  nonrespondents. 

Relationship  to  Mail  Survey  Method.  A few  studies  have  related 
response  bias  to  survey  method.  The  ethnicity  (Friedman  & Goldstein, 

1975)  and  sex  (Field,  1975)  of  the  signature  on  the  cover  letter  were 
found  to  affect  neither  rate  of  return  nor  response  bias  on  attitude  items. 

Nevin  and  Ford  (1976)  studied  the  effects  of  a follow-up  that  in- 
cluded a veiled  threat  (that  the  nonrespondent  would  continue  receiving 
such  letters  until  he  responds)  or  a casual  letter.  The  response  rate 
for  the  "veiled  threat"  group  was  significantly  higher  than  for  the  "casual 
letter"  group,  but  the  researchers  found  no  response  bias  on  the  question- 
naire concerning  residents'  attitudes  about  university  residence  halls. 

Panos  and  Rice  (1967)  used  a 2 x 2 x 3 design  (type  of  cover  letter 
x type  of  outgoing  postage  x class  of  outgoing  mail).  They  found  signi- 
ficant main  effects  on  response  rate  but  not  on  the  measures — mostly  apti- 
tude variables — obtained  from  the  respondents. 

Correcting  for  Response  Bias 

Four  methods  have  been  proposed  to  correct  for  possible  response 
bias:  (1)  using  population  data,  (2)  obtaining  a subsample  with  a 100 

percent  return  rate,  (3)  using  early  and  late  respondents,  and  (4)  relating 
the  probability  of  responding  to  actual  response. 

Using  Population  Data.  By  comparing  the  respondents  and  non- 
respondents on  dimensions  for  which  complete  data  are  available  for  the 
entire  population,  weighting  schemes  can  be  developed  for  making  the  re- 
spondents representative  of  the  population.  Mandell  (1974)  presented  a 
method  for  determining  the  bias  in  the  estimate  of  a population  mem  as 
a function  of  differential  response  rates  among  subgroups,  based  on  back- 
ground variables.  Rubin  (1975)  developed  a method  (based  on  Bayesian 
techniques)  that  estimates  the  effect  of  nonrespondents,  using  the  linear 
relationship  of  background  variables  to  observed  variables.  It  requires 
the  assignment  of  two  subjective  parameters  that  summarize  the  researcher's 
notions  about  the  similarity  of  these  relationships  for  respondents  and 
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nonrespondents.  However,  weighting  nonrespondents  according  to  the 
similarity  of  their  biographical  characteristics  to  those  of  the  respon- 
dents does  not  necessarily  correct  the  sample's  biased  composition. 

There  are  dangers  in  using  this  method.  An  absence  of  bias  in 
the  background  variables  does  not  imply  an  absence  of  bias  in  the  variables 
measured  on  the  survey  (Donald,  1960;  Ford  & Zeisel,  1949).  Similarly, 
when  differences  are  observed  in  background  variables,  it  is  not  possible 
to  determine  whether  the  source  of  this  difference  is  due  to  nonresponse 
or  to  other  aspects  of  sampling  (Lowe  & McCormick,  1955). 

Obtaining  a Subsample  with  100  Percent  Return  Rate.  Drawing  a 
subsample  of  the  nonrespondents  and  making  every  effort  to  obtain  responses 
is  proposed  by  several  methodologists.  Birnbaum  and  Sirhew  (1950),  Fuller 
(1974),  and  Hansen  and  Hurwitz  (1946)  present  methods  for  correcting  the 
estimates  based  on  the  responses  from  the  subsample.  F.l-Badrv  (1956) 
proposed  a variation  of  this  method  in  which,  using  a stepwise  procedure, 
a random  sample  of  nonrespondents  at  each  wave  receives  a follow-up 
(questionnaire  or  other  methods),  with  the  lust  wave  consisting  of  the  most 
resistant  subjects.  Each  method  assumes  that  a 100  percent  subsample  can 
eventually  be  obtained,  an  unrealistic  assumption  in  most  applied  settings. 

Using  Early  and  Late  Respondents.  This  method  of  correcting  for 
response  bias  (1)  assumes  that  the  tendency  to  respond  lies  on  a continuum, 
and  (2)  utilizes  data  from  the  late  respondents  to  estimate  information  on 
the  nonrespondents.  Methodologists  who  have  described  this  approach  assume 
either  that  the  nonrespondents  resemble  the  late  respondents  or  that  non- 
respondents are  more  extreme  that  late  respondents  (Bridge,  1974;  Clausen  4 
Ford;  1947;  Deming,  1953;  Donald,  1960;  Edgerton,  Britt,  & Norman,  1947; 
Ellis,  Endo,  6 Armer,  1970;  Ferber,  1948;  Filion,  1974;  Franzen  & Lazarsfeld, 
1945;  Hilgard  & Payne,  1944;  Kivlin,  1965;  Larson  6 Catton,  1959;  Lehman, 
1963;  Lundberg  & Larsen,  1949;  Mayer  & Pratt,  1966;  Newman,  1962;  Pace, 

1939;  Reid,  1942;  Robins,  1963;  Scheverian  4 Blaine,  1966;  Shuttleworth, 

1939,  1940;  Suchman  & McCandless,  1940;  Toops,  1926). 

If  the  assumptions  are  valid,  this  method  may  make  it  possible  to 
accurately  estimate  the  population  parameters.  However,  past  studies 
generally  fail  to  support  the  assumption  that  nonrespondents  are  more  like 
late  respondents  than  early  respondents  (Baur,  1947;  Eckland,  1965;  Ellis, 
Endo,  & Armer,  1970;  Filion,  1974;  Ford  & Zeisel,  1949;  Kivlin,  1965; 

Mayer  4 Pratt,  1966;  Pace,  1939;  Scott,  1961). 

Filion  (1974)  fit  a linear  regression  over  cumulative  response 
waves  to  examine  the  response  bias  in  the  Canada  Migratory  Game  Bird 
Harvest  Survey.  He  found  good  linear  fits  to  three  waves  of  responses  on 
behavioral  measures  (e.g.,  proportion  of  permit  holders  who  hunted  at 
least  one  day),  but  these  measures  changed  very  little  with  the  addition 
of  respondents  at  each  wave. 

Eckland  (1965)  found  that  college  dropouts  required  less  prodding 
to  respond  than  graduates,  but  that  there  was  no  relationship  between  the 
amount  of  prodding  required  and  the  veracity  of  respondents'  replies. 
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Clausen  and  Ford  (1947)  found  that  educational  plans  changed  mono- 
tonically  from  first-  to  second-  to  third-wave  respondents.  However,  the 
relationship  does  not  appear  to  be  linear. 

Ellis,  Endo,  and  Armer  (1970)  explored  trends  in  their  measure- 
ments by  dividing  their  target  population  into  early  respondents,  late 
respondents  (respond  only  after  one  follow-up),  potential  nonrespondents 
(resisting  normal  follow-up  but  responding  to  special  efforts),  and  non- 
respondents. They  found  no  evidence  of  a linear  trend  on  demographic 
variables.  On  three  academic  potential  variables,  they  found  monotonically 
decreasing  curves,  but  quantitative  Scholastic  Aptitude  Test  was  the  only 
one  for  which  linear  extrapolation  would  have  been  effective.  On  four 
academic  performance  variables,  no  fits  appear  to  be  linear  and  only 
one  is  monotonic.  In  all  seven  academic  variables,  the  potential  respon- 
dents resembled  the  late  respondents  more  than  they  resembled  the  nonre- 
spondents, which  runs  counter  to  the  assumptions  of  several  schemes  for 
correcting  for  nonrespondents.  Thus,  although  there  was  no  bias  on  demo- 
graphic variables,  there  was  a bias  on  academic  potential  variables,  and 
the  direction,  although  not  the  magnitude,  could  generally  be  predicted. 

Relating  the  Probability  of  Responding  to  Actual  Response.  Tsai 
(1973)  developed  a method  that  uses  two  or  three  mailings  of  surveys, 
different  in  content  except  for  the  item(s)  on  which  a response  bias  is 
being  estimated.  All  mailings  are  sent  to  a random  sample  of  the  popula- 
tion, and  the  number  of  surveys  responded  to  indicates  the  probability 
of  responding.  A functional  relationship  between  probability  of  responding 
and  actual  response  is  found  from  which  the  population  mean  may  be  estimated. 
As  d escribed  in  the  Response  Bias  section,  this  method  was  employed  by 
Broedling  and  Fuller  (1976)  who  found  no  relationship  between  tendency 
to  respond  and  attitudinal  measures. 

None  of  the  methods  available  for  correcting  for  response  bias  is 
free  from  inherent  difficulties:  (1)  demographic  variables  may  indicate 

a difference  when  none  is  present,  and  vice  versa,  (2)  a subsample  with 
100  percent  return  rate  is  extremely  difficult  to  obtain,  (3)  nonrespon- 
dents are  not  always  similar  to  late  respondents,  and  (4)  tendency  to 
respond  may  not  be  related  to  actual  responses. 

Reactance  Theory 

Brehm  (1966)  postulated  that  each  individual  has  a set  of  behavioral 
freedoms  and  that,  if  any  of  these  freedoms  is  threatened,  the  individual 
will  react  and  become  motivated  to  reestablish  his  freedom.  Social  influence, 
or  persuasive  communication,  has  been  studied  extensively  as  one  major 
determinant  of  reactance.  Typically,  laboratory  experiments  have  manipulated 
social  influence  by  having  confederates  restrict  the  subjects'  freedom  of 
choice  through  freedom-threatening  statements  (Brehm  & Sensenig,  1966; 

Worchel  & Brehm,  1971).  The  importance  of  the  threatened  freedom  to  the 
subject  and  whether  or  not  the  subject  was  able  to  attribute  causality  for 
the  behavior  to  environmental  Influences  were  found  to  affect  the  degree 
of  reactance  experienced  (Worchel  & Andreoli,  1974). 
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Reactance  has  also  been  used  as  an  explanation  for  an  individual's 
unwillingness  to  help  others.  Berkowitz  (1973)  contended  that  a demand 
(explicit  or  implicit)  to  help  someone,  and  even  a felt  obligation,  is  a 
bothersome  threat  to  the  individual 's  freedom  of  action.  He  cites  evidence 
that,  at  times,  increased  pressure  to  aid  a person  in  need  reduces  the  in- 
dividual's willingness  to  help  the  one  who  is  dependent  upon  him. 

Organ  (1974)  found  that  conferral  of  a favor  by  a superior  on  a 
suborinate  will  generate  counter-compliance  on  the  part  of  the  subordinate 
in  a situation  which  restricts  the  subject's  range  of  freedom  (e.g.,  a high 
degree  of  surveillance).  Mazis  (1975)  cites  other  factors  that  have  been 
found  to  affect  reactance:  conformity  pressures,  the  involvement  of  peers 

in  eliminating  alternatives,  and  individual  difference  variables  (including 
feelings  of  competence,  feelings  of  inadequacy,  and  locus  of  control). 

Mazis  (1975),  in  a study  of  housewives'  responses  to  implementation 
of  an  antiphosphate  ordinance,  found  that  the  principal  mode  of  response  was 
toward  reduced  attractiveness  of  the  forced  alternative. 

These  findings  could  be  extended  to  the  situation  encountered  in 
mail  surveys.  The  more  the  subject's  freedom  is  restricted  by  pressure 
to  respond  to  the  survey,  the  more  likely  he  is  to  negatively  react  toward 
the  survey.  His  feelings  could  be  expressed  directly  by  not  responding  to 
the  survey,  or  indirectly  in  terms  of  the  attitudes  expressed  on  the  topic 
of  the  survey. 
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METHOD 


Sample 

This  study  was  conducted  in  1975  as  part  of  the  Department  of  Defense 
Family  Housing  Preference  Survey  (Stumpf  & Kieckhaefer,  1975).  A question- 
naire was  used  to  obtain  both  demographic  and  attitudinal  data  from  military 
families.  The  questionnaire  was  mailed  to  a systematic  sample  and  stratified 
by  paygrade  group,1  with  disproportionate  sampling  of  the  various  groups 
(Table  1).  Selection  for  the  survey  was  limited  to  military  personnel 
having  dependents  who  either  lived  with  them  or  were  primarily  supported 
by  them. 

The  follow-up  study  was  confined  to  the  San  Diego  portion  of  the  survey 
sample.  Approximately  1500  Navy  and  Marine  Corps  personnel  were  chosen  at 
random  and  assigned  to  10  follow-up  groups. 

Since  the  sampling  was  disproportionate  by  paygrade,  the  selectees 
within  each  of  the  10  groups  were  weighted  by  paygrade  group  in  all  analyses, 
based  upon  the  population  of  personnel  meeting  the  criteria  for  possible 
inclusion  in  the  survey  as  reported  in  the  Navy  and  Marine  Corps  personnel 
strength  reports  as  of  31  January  1975  (Table  2). 

Design  and  Data  Collection 

Each  individual  selected  for  inclusion  in  the  survey  was  sent  a package 
of  materials  in  a government-franked  envelope  marked  "first  class"  and 
mailed  to  his  duty  station  address.  This  package  included  the  following 
items: 


1.  A cover  letter  signed  by  the  Deputy  Assistant  Secretary  of  Defense. 

2.  A military  questionnaire  containing  107  multiple-choice  items  and 
an  optical  scan  answer  sheet. 

3.  A spouse  questionnaire  containing  87  multiple-choice  items  and  an 
optical  scan  answer  sheet. 

4.  A Family  Housing  questionnaire  containing  20  items  (some  write-in 
and  some  multiple  choice)  with  two  separate  instruction  sheets. 

5.  A franked  return  envelope. 

Ail  variables  included  in  this  study  were  obtained  from  the  military 
questionnaire  (item  2 above). 


*The  strata  were  (1)  E-l  to  E-3,  (2)  E-4  to  E-6,  (3)  E-7  to  E-9, 
(4)  WO-1  to  WO-4  and  0-1  to  0-3,  (5)  0-4  to  0-5,  and  (6)  0-6. 
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Table  1 

Survey  Population  by  Paygrade  Stratum  and  Follow-up  Group 


Follow-up 

Group 

Paygrade 

Stratum 

1 

2 

3 

4 

5 

6 

- Group 
Total 

I 

19 

18 

16 

18 

18 

7 

96 

II 

21 

18 

15 

17 

18 

7 

96 

III 

21 

17 

17 

17 

16 

8 

96 

IV 

21 

14 

17 

18 

17 

9 

96 

V 

23 

16 

15 

18 

16 

8 

96 

VI 

21 

14 

17 

18 

17 

9 

96 

VII 

21 

16 

14 

17 

19 

9 

96 

VIII 

21 

17 

17 

19 

15 

7 

96 

IX 

22 

17 

16 

17 

16 

8 

96 

X 

125 

106 

99 

98 

99 

65 

592 

‘ 

Stratum 

Total 

315 

253 

243 

257 

251 

137 

1456 

Target 

Population  4983  22826 


6387 


3398 


2055 


294 


Table  2 


Subject  Weighting  by  Paygrade  Stratum  and  Follow-up  Group 


Paygrade  Stratum 

Group 

1 

2 

3 

4 

5 

6 

I 

.63 

3.05 

.96 

.45 

.27 

.10 

II 

.57 

3.05 

1.02 

.48 

.27 

.10 

III 

.57 

3.23 

.90 

.48 

.31 

.08 

IV 

.57 

3.92 

.90 

.45 

.29 

.08 

V 

.52 

3.43 

1.02 

.45 

.31 

.08 

VI 

.57 

3.92 

.90 

.45 

.29 

.08 

VII 

.57 

3.43 

1.10 

.48 

.26 

.03 

VIII 

.57 

3.23 

.90 

.43 

.33 

.10 

IX 

.55 

3.23 

.96 

.48 

.31 

.08 

X 

.59 

3.19 

.96 

.51 

.31 

.07 

Note : 


ii  n , 

For  paygrade  i,  w - — . — 

— in  n^ 

where  n » sample  size,  N ■ population  size,  n^  = size  of  i.th 
sample  stratum,  and  - size  of  ^th  population  stratum. 
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After  the  original  nailing,  follow-up  treatment  varied  among  the  groups. 
A completely  crossed  three-factor  design  with  two  levels  for  each  factor 
was  employed.  The  three  factors  designated  (1)  whether  the  follow-up  was 
sent  by  regular  mail  or  by  special  delivery,  (2)  whether  the  selectee  was 
mailed  only  a letter  in  the  follow-up(s)  or  a letter  with  duplicates  of 
the  original  materials,  and  (3)  whether  one  or  two  follow-ups  were  sent. 

In  addition,  one  group  received  no  follow-up  and  one  group  received  as 
extensive  a follow-up  as  possible  (after  receiving  two  special  delivery 
package  follow-ups),  including  phone  calls  and  additional  mailings. 

Each  member  of  Group  I was  sent  one  follow-up  letter  by  regular  mail; 
Group  IT,  one  by  special  delivery;  Group  III,  two  letters  by  regular  mail;  • 
and  Group  IV,  two  by  special  delivery.  Each  member  of  Group  V was  sent 
one  follow-up  package  by  regular  mail;  Group  VI,  one  by  special  delivery; 
Group  VII,  two  packages  by  regular  mail;  and  Group  VIII,  two  by  special 
delivery.  Members  of  Group  IX  received  extensive  follow-up  and  Group  X, 
no  follow-up.  (Groups  I-VIII  are  subsequently  referred  to  as  "treatment 
groups.") 

The  first  follow-ups  were  mailed  to  nonrespondents  approximately  10 
days  after  the  original  mailing,  and  the  second  to  those  who  still  had  not 
responded  about  1 week  later.  No  further  attempts  were  made  to  contact 
selectees  in  Groups  I-VIII  or  Group  X,  but  any  responses  received  from  these 
groups  up  to  a year  after  the  original  mailing  were  included  in  the  analysis. 

The  analyses  of  the  12  variables  employed  are  based  upon  the  usable 
responses  obtained  from  those  military  personnel  who  were  accompanied  by 
their  dependents  and  were  still  stationed  in  San  Diego  when  they  com- 
pleted the  survey  (Table  3).  About  5 percent  of  all  those  responding  to 
the  survey  were  instructed  not  to  provide  any  attitudinal  data  because 
they  were  not  living  with  their  dependents  at  the  time  of  the  survey. 

These  Individuals  were  included  in  the  computation  of  return  rates  but 
not  in  the  analyses  of  response  bias. 

Additional  respondent  losses  were  experienced  due  to  ship  deployments, 
frequent  transfers  to  other  duty  stations,  and  omission  of  the  respondent's 
social  security  number,  which  was  required  to  identify  which  treatment  group 
he  belonged  to.  Although  only  a small  percentage  of  the  obtained  responses 
were  unusable,  the  actual  return  rates  would  be  higher  than  reported  herein 
if  the  unreachable  and  unidentifiable  individuals  had  not  heen  considered 
part  of  the  selected  sample.  However,  it  was  decided  not  to  distinguish 
between  unavallables,  unwilling  nonrespondents,  and  the  small  number  of 
respondents  who  refused  to  Identify  themselves,  since  the  study  was  designed 
to  compare  the  obtained,  usable  responses. 
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Table  3 


Number  of  Subjects  (with  Dependents)  Who  Responded 


Follow-up 

Group 

Paygrade 

Stratum 

Group 

Total 

1 

2 

3 

4 

5 

6 

I 

2 

8 

11 

9 

8 

5 

43 

II 

4 

9 

10 

7 

10 

3 

43 

III 

3 

10 

9 

9 

10 

5 

46 

IV 

3 

10 

13 

11 

12 

6 

55 

V 

4 

6 

10 

7 

11 

5 

43 

VI 

3 

8 

5 

7 

8 

5 

36 

VII 

4 

7 

7 

8 

11 

7 

44 

VIII 

1 

10 

7 

7 

6 

4 

35 

IX 

4 

9 

11 

16 

13 

6 

59 

X 

6 

24 

31 

30 

49 

38 

178 

Stratum 

Total 

34 

101 

114 

111 

138 

84 

582 

The  primary  concern  was  not  with  the  specific  factor  motivating  the 
Individual  selectee  to  respond  but,  rather,  with  a comparison  of  the 
different  follow-up  methods  to  see  whether,  as  a whole,  the  responses 
obtained  using  one  follow-up  method  differ  from  those  obtained  using  other 
follow-up  methods.  Thus  Groups  I-IX  are  composed  of  a variety  of  respondent 
types  (e.g.,  the  respondents  assigned  to  a group  receiving  two  follow-ups 
are  those  who  responded  without  any  follow-up,  those  who  responded  after 
one,  and  those  who  required  two  before  responding).  In  other  words,  the 
"two  follow-ups"  groups  are  typical  of  the  total  sample  of  responses  one 
would  be  likely  to  obtain  using  a mail  survey  with  two  follow-ups. 

Description  of  Variables 

The  10  groups  were  compared  with  regard  to  (1)  return  rate  and  (2)  12 
selected  variables,  categorized  as  measures  of  preference,  demographics. 
Importance,  satisfaction,  and  opinions  (see  Table  4).  Demographic  and 
attltudinal  variables  were  selected  from  the  large  number  of  items  on 
the  questionnaire,  based  on  two  criteria:  (1)  no  pair  of  variables 

should  yield  a Pearson  product  moment  correlation  coefficient  less  than 
.50  (to  avoid  redundancy),  and  (2)  the  selected  variables  should  repre- 
sent a broad  variety  of  the  variable  types  likely  to  be  found  in  survey 
research. 


Table  4 


Description  of  Variables  Used  in  Study 


Category 

Variable 

Number 

Variable  Name 

High  Score  Indicates 

Preference 

Measures 

1 

Preference  for  civilian 
community 

Prefers  civilian'3 
community 

2 

Preference  for  military 
career 

Prefers  military 
career 

Demographic 

Measures 

3 

Number  of  dependents 
you  live  with 

Large  number  of 
dependents 

4 

Paygrade  group 

High  paygrade 

5 

Housing  location 

Occupies  civilian 
housing 

Importance 

Measure 

6 

Perceived  importance  of 
various  military  benefits 
and  facilities 

Low  importance'3 

Sat isfactlon 
Measures 

7 

Satisfaction  with  quality 
of  life 

High  satisfaction'3 

8 

Satisfaction  with  amount 
of  housing  allowance  for 
paygrade 

High  satisfaction 

9 

Satisfaction  with  current 
housing  cost s 

High  satisfaction 

Opinion 

Measures 

10 

Opinion  of  present  policy 
for  building  military 
family  housing 

Favorable  opinion 

11 

Opinion  of  variable 
housing  allowance  policy 

Favorable  opinion 

12 

Opinion  of  fair  market 
rental  policy 

Favorable  opinion 

aMoblle  home  occupants  were  excluded  from  this  measure. 
^Multiple  item  scaled  score. 
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Hypotheses  and  Vr edict  Ions 


Return  Rate 


Higher  pressure  groups  will  yield  higher  return  rates  than  lower 
pressure  groups.  Specifically: 

1.  One  follow-up  will  yield  a higher  return  rate  than  no  follow- 
up, and  two  will  yield  a higher  rate  than  one. 

2.  Extensive  follow-up  will  produce  the  highest  return  rate. 

3.  Use  of  special  delivery  mail  will  yield  a higher  return  rate 
than  regular  mail. 

4.  A package  of  duplicate  materials  will  result  in  a higher  return 
rate  than  a letter  alone. 

Response  Bias 

It  was  expected  that  those  with  the  greatest  interest  in  the  survey 
topic  or  the  sponsoring  organization  and  those  holding  the  most  extreme 
opinions  on  the  survey  topic  would  be  the  two  groups  most  likely  to  respond. 
Two  types  of  predictions  arise: 

1.  On  variables  1-6  (preference,  demographic,  and  importance),  the 
groups  receiving  the  greatest  pressure  to  respond  would  have  different  mean 
scores  than  those  receiving  less  pressure.  Specifically,  higher  pressure 
groups  would  score  higher  on  variables  1,  5,  and  6,  and  lower  on  variables 
2,  3,  and  4,  based  on  past  research  on  demographic  variables  (Broedling  & 
Fuller,  1976)  and  the  expectation  that  those  involved  in  the  military  com- 
munity would  be  more  likely  to  respond. 

2.  On  variables  7-12  (satisfaction  and  opinion),  respondents  in 
the  groups  receiving  the  greatest  amount  of  pressure  would  have  less  extreme 
opinions  and,  therefore,  smaller  group  standard  deviations,  when  compared 
with  those  receiving  less  pressure  to  respond. 

Reactance  Theory 

On  the  basis  of  reactance  theory,  it  was  predicted  that  the  higher 
pressure  groups  would  react  more  negatively  to  the  preference,  attitude, 
satisfaction,  and  opinion  variables.  Specifically,  they  would  be  expected 
to  have  (1)  higher  mean  scores  on  variables  1 and  6 and  (2)  lower  mean 
scores  on  variables  2 and  7-12.  It  should  be  noted  that  some  of  these  pre- 
dictions differ  from  those  based  on  response  bias  research.  The  extent  to 
which  each  set  of  predictions  is  supported  will  be  discussed  in  Results. 

Method  of  Analysis 

Several  analyses  were  performed  on  the  responses  from  the  10  groups, 
including  a three-way  analysis  of  variance  (ANOVA)  on  the  8 treatment  groups 
and  two  one-way  ANOVAs  using  linear  contrasts.  The  first  contrast  compared 


the  mean  of  the  treatment  groups  with  that  of  the  extensive  follow-up  groups 
and  the  second,  the  mean  of  the  treatment  groups  with  that  of  the  control 
group.  Because  a large  number  of  statistical  tests  were  performed  on  the 
data,  the  alpha  level  of  significance  was  set  at  .01.  Since  the  design 
used  disproportionate  sampling,  the  weights  appearing  in  Table  2 were 
applied  to  all  observations.  Let  w^  be  the  weight  for  person  i and  x 

be  the  response  from  person  i on  item  j.  Then,  for  example,  the  estimate 
of  the  mean  for  item  j is 


The  return  rate  was  computed  similarly  for  all  selectees  in  a given  group, 
assigning  a value  of  1 to  x if  the  person  responded  and  0 if  the  person 
did  not  respond. 
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RESULTS  AND  DISCUSSION 


Return  Rate 


Using  the  techniques  described  previously,  return  rates  for  the  10 
groups  were  compared  (Tables  5-7).  A t-test  comparing  the  mean  of  the 
treatment  groups  with  that  of  the  no  follow-up  group  resulted  in  a signi- 
ficant difference  (t  - 5.10,  p < .01),  indicating  that  using  some  follow- 
up procedure  will  yield  a far  greater  return  rate  than  using  none  at  all. 

A similar  t-test,  which  compared  the  mean  of  the  treatment  groups  with  that 
of  the  extensive  follow-up  group,  also  resulted  in  a significant  difference 
(t  * U. 76,  p < .01),  indicating  tliat  still ^greater  return  rates  can  be 
achieved  through  extensive  follow-up.  However,  the  three-way  analysis 
of  variance  on  the  eight  treatment  groups  indicated  that  the  type  of  mail, 
the  number  of  follow-ups,  and  the  type  of  follow-up  material  (letter  vs. 
pac’age)  did  not  affect  the  return  rate;  i.e.,  none  of  the  main  effects 
or  Interactions  for  these  factors  were  significant  at  the  .01  level. 

Certain  aspects  of  the  procedures  used  may  account  for  the  non- 
significant differences  in  return  rate.  First,  the  "special  delivery" 
follow-ups  may  have  been  distributed  simultaneously  with  the  regular  mail 
distributed  at  the  duty  station.  Thus,  while  the  selectee  might  be  impressed 
by  the  "special  delivery"  stamp  on  the  envelope,  it  would  not  be  likely  to 
have  the  same  impact  as  if  the  letter  or  package  had  been  specially  delivered 
to  his  residence.  Second,  the  fact  that  the  first  and  second  follow-ups 
were  very  closely  spaced  (1  week  apart)  may  have  lessened  the  impact  of  the 
second  follow-up.  Third,  the  "letter  vs.  package"  comparison  may  have  been 
affected  by  two  offsetting  tendencies:  while  the  follow-up  package  may 

have  encouraged  responses  from  some  selectees  who  misplaced  or  discarded  the 
original  package,  the  large  number  of  items  included  in  the  follow-up  package 
may  have  guaranteed  reactance  among  other  selectees  and  made  them  less 
likely  to  respond  than  those  receiving  only  a letter  follow-up. 

Response  Bias 

Measures  of  Central  Tendency 

Group  means  are  shown  in  Table  8.  Linear  contrasts  between  the  no 
follow-up  and  treatment  groups  and  the  treatment  and  extensive  follow-up 
groups  indicated  that  none  of  the  differences  in  any  of  the  six  variables 
for  which  mean  differences  were  predicted  (variables  1-6)  were  statistically 
significant  at  the  .01  level,  despite  the  fact  that  return  rates  differed 
substantially  (ranging  from  .26  to  .77). 
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Table  5 


Return  Rate  for  Each  Follow-up  Method 


One  Follow-up 

Two  Follow-ups 

Follow-up 

Regular  Spec.  Del. 

Regular 

Spec.  Del. 

better 

Package 

.46  .50 

.46  .43 

.52 

.43 

.61 

.46 

Table  6 

Mean  Return  Rate  by  Number  of 

Follow-ups 

Number  of 

Response 

Number  of 

Fol low-ups 

Rate 

Selectees 

None 

.26 

592 

One 

.46 

384 

Two 

.50 

384 

Extensive 

.77 

96 

Mean  Return  Rate  by  Type 

Table  7 

of  Mail  and  by  Type 

of  Material 

Follow-up 

Response 

Number  of 

Method 

Rate 

Selectees 

better  Follow-up 

.52 

384 

Package  Follow-up 

.44 

384 

Regular  Mall 

.47 

384 

Special  Delivery 

.50 

384 

Weighted  Means  on  Variables  1-12  for  Each  Follow-up  Method 
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Table  9 shows  that  the  results  of  the  three-way  ANOVAS  on  the  treat- 
ment groups  are  inconsistent  from  one  variable  to  another  and  that  little 
support  is  provided  for  the  differences  between  group  means  that  were  pre- 
dicted on  the  basis  of  response  bias  research.  Of  the  six  variables  that 
were  expected  to  show  differences  in  means  due  to  response  bias  (1-6),  four 
9howed  neither  significant  main  effects  nor  interaction  effects  (variables 
1-4).  However,  variable  6 (importance  of  military  benefits  and  facilities) 
showed  significant  main  effects  for  type  of  material  (B)  and  type  of  mail 
(A).  As  predicted,  those  receiving  packages  scored  higher  than  those 
receiving  letters  alone  (mean  of  3.9  vs.  3.6).  Contrary  to  prediction, 
however,  those  receiving  special  delivery  mail  scored  lower  than  those 
receiving  regular  mail  (mean  of  3.6  vs.  3.9).  There  were  no  main  effects 
for  variable  5 (housing  location),  but  there  were  interaction  effects  for 
type  of  mail  by  type  of  material  (A  x B)  and  type  of  mail  by  number  of 
mailings  (Ax  C) . As  predicted,  when  regular  mail  was  used,  the  "package" 
groups  scored  higher  than  the  "letter"  groups,  but  the  opposite  was  true 
when  special  delivery  mail  was  used.  Also  as  predicted,  when  two  follow-ups 
were  administered,  "special  delivery"  groups  scored  higher  than  the  "regular 
mail"  groups,  but  the  groups  receiving  only  one  follow-up  showed  the  opposite 
effect . 


The  failure  to  find  significant  support  for  the  predicted  differences 
in  means  among  the  eight  treatment  groups  might  be  attributable  to  the 
design  features  discussed  above  in  Return  Rate.  However,  these  design 
features  cannot  account  for  the  finding  of  no  significant  difference  on 
the  first  six  variables  when  the  two  most  extreme  groups  (no  follow-up 
and  extensive  follow-up)  were  compared  post  hoc,  nor  can  they  explain 
the  lack  of  significant  differences  when  the  eight  treatment  groups  were 
compared  with  the  control  group  and  the  extensive  follow-up  group. 
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Table  9 


Summary  of  Analyaea  of  Variance  for  Variables  1-12 


Main  Effects 

Interaction  Effects 

Variable  Number  and  Name 

A - 

Type 

of 

Mail 

B - 

Type 

of 

Material 

c - 

Number 

of 

Mailings 

AXB 

AXC  BXC  AXBXC 

1. 

Preference  for  civilian 
community 

2. 

Preference  for  military 

career 

3. 

Number  of  dependents 
you  live  with 

4. 

Paygrade  group 

5. 

Housing  location 

X 

X 

6. 

Perceived  importance  of 
various  benefits  and 
facilities 

X 

X 

7. 

Satisfaction  with  quality 
of  life 

X 

X 

8. 

Satisfaction  with  amount 
of  housing  allowance 

X 

9. 

Satisfaction  with  current 
bagging  co«te 

10. 

Opinion  oi  present  policy 
for  building  military 
housing 

X 

11. 

Opinion  of  variable 
housing  allowance  policy 

X 

12. 

Opinion  ol  fair  marker 
rental  policy 

X 

Note:  Effects  that  were  significant  at  the  .01  level  are  indicated  by  "X." 

Blank  cells  indicate  that  effect  was  not  significant  at  .01  level. 
Range  of  weighted  sample  size  287<n<365 
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No  predictions  were  made  as  to  the  effect  of  response  bias  on  the 
group  means  for  the  remaining  six  variables  (7-12).  The  linear  contrasts 
revealed  two  statistically  significant  differences  for  these  variables 
(8  and  12)  between  the  no  follow-up  and  treatment  groups  only.  There 
were  no  significant  differences  using  a post  hoc  linear  contrast  between  the 
no  follow-up  and  extensive  follow-up  groups,  perhaps  due  to  the  smaller  number 
of  observations  in  the  latter.  The  three-way  ANOVAS  summarized  in  Table 
9,  however,  showed  significant  main  effects  for  variables  7 and  10  and  sig- 
nificant interaction  effects  for  variables  7,  8,  11,  and  12.  As  shown: 

1.  Variables  7 and  10  indicated  a significant  main  effect  for  type 
of  material  (B) , with  "letter"  respondents  scoring  higher  than  "package" 
respondents. 

2.  Variable  8 had  a significant  type  of  mail  by  type  of  material 
interaction  (Ax  B) , with  "regular  mail"  groups  scoring  higher  with  a letter 
than  a package,  while  "special  delivery"  groups  scored  higher  with  a package. 

1.  Variable  7 yielded  a significant  interaction  effect  for  type 
of  mail  by  number  of  mailings  (A  x C) . The  "one  follow-up"  group  scored 
lower  when  the  follow-up  was  delivered  by  special  delivery  mail  than  when 
it  was  delivered  by  regular  mail,  while  the  "two  follow-ups"  groups  scored 
lower  with  regular  mail. 

4.  Variable  11  was  the  only  one  producing  a significant  type  of 
material  by  number  of  mailings  interaction  (B  x C).  The  "letter"  groups 
scored  lower  than  the  "package"  groups  when  one  follow-up  was  mailed,  but 
with  two  follow-ups  the  letter  groups  scored  higher. 

5.  Variable  12  was  the  only  one  to  produce  a significant  three-way 
interaction  (A  x B x C) . The  "letter,  special  delivery,  one  follow-up" 
group  scored  much  lower  than  the  others,  while  the  "package,  regular,  two 
follow-ups"  group  and  the  "letter,  special  delivery,  two  follow-ups"  group 
scored  much  higher. 

One  method  of  extrapolating  to  predict  nonresponse  bias  establishes 
trends  from  the  results  of  several  waves  and  predicts  the  nonresponses  by 
continuing  these  trends  (Ferber,  1948;  Ford  & Zeisel,  1949;  Campbell,  1949). 
However,  the  trends  may  not  be  linear  or  even  monotonic  (Baur,  1947).  Two 
variables  in  this  study  (8  and  12)  indicated  significant  differences  between 
the  no  follow-up  and  treatment  groups.  The  means  indicate  that  the  treatment 
and  extensive  follow-up  groups  are  very  similar  for  these  two  variables. 

Measures  of  Dispersion 

If  a response  bias  exists,  it  may  manifest  itself  in  the  spread  of 
responses  about  the  mean  rather  than  in  differences  between  group  means. 

It  was  predicted  that  response  bias  would  cause  those  who  required  more 
pressure  before  responding  to  be  more  neutral  on  the  three  satisfaction 
variables  (7,  8,  and  9)  and  the  three  policy  preference  variables  (10,  11, 
and  12).  In  other  words,  selectees  who  were  extremely  satisfied  or  dis- 
satisfied and  selectees  who  had  strong  feelings  for  or  against  housing 
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policies  would  tend  to  respond  initially  to  the  survey,  whereas  the  more 
neutral  selectees  would  decline  from  responding  unless  additional  pressure 
was  applied  to  them.  The  F-ratios  indicated  no  significant  differences 
between  the  variances  of  the  no  follow-up,  treatment,  and  extensive  follow- 
up groups,  indicating  that  low  pressure  respondents  do  not  necessarily  hold 
more  extreme  opinions  than  high  pressure  respondents. 

Reactance  Theory 

It  was  predicted  on  the  basis  of  reactance  theory  that  higher  pressure 
groups  would  have  higher  mean  scores  on  variables  1 and  6 and  lower  mean 
scores  on  variables  2 and  7-12.  The  linear  contrasts  indicated  that,  al- 
though the  mean  of  the  no  follow-up  group  was  different  from  that  of  the 
treatment  groups,  it  increased  with  increasing  follow-up  on  variable  8 and 
decreased  on  variable  12.  The  significant  main  effects  attributable  to 
type  of  material  (letter  vs.  package)  were  all  in  the  predicted  direction, 
with  the  package  groups  (i.e.,  those  receiving  more  pressure  to  respond) 
scoring  more  negatively  than  the  letter  groups.  The  only  other  significant 
main  effect  was  in  the  opposite  direction  of  that  predicted  by  reactance 
theory,  with  the  special  delivery  groups  scoring  less  negatively  than  the 
regular  mail  groups.  This  may  indicate  that  the  duplicate  package  of 
materials  had  a stronger  psychological  impact  upon  the  selectee  than  did 
a special  delivery  follow-up  or  an  additional  follow-up.  The  interaction 
effects  appear  inconsistent,  as  in  the  analysis  of  response  bias. 


CONCLUSIONS 


The  results  of  past  studies  indicating  that  some  type  of  follow-up 
procedure  results  in  greater  return  rates  than  no  follow-up  have  been 
supported  in  this  study.  The  study  also  indicated  that  extensive  follow- 
up (including  phone  contact)  substantially  improves  return  rates  compared 
with  less  extensive  follow-up  methods.  However,  it  was  not  shown  that  one 
of  the  follow-up  techniques  used  in  this  study  increases  return  rate  more 
effectively  than  another,  unless  extensive  follow-up  procedures  are  used. 

Although  return  rates  between  the  control,  treatment,  and  extensive 
follow-up  groups  were  shown  to  differ,  only  the  no  follow-up  and  treatment 
groups  differed  on  the  population  parameter  estimates  of  the  mean,  and 
the  differences  were  evidenced  on  only  2 of  the  12  variables  included  in 
the  study  (neither  of  which  was  predicted  based  on  past  research  or  intui- 
tion). The  12  variables  showed  few  consistent  main  effects  or  interaction 
effects  in  the  analyses  of  variance,  and  no  marked  differences  in  means 
were  found  when  extensive  follow-up  procedures  were  applied. 

The  main  effect  that  was  statistically  significant  for  the  largest 
number  of  variables  was  the  type  of  material  used  in  the  follow-up — letter 
or  duplicate  questionnaire  package.  The  effect  was  found  for  three  vari- 
ables, with  the  "package"  groups  reacting  more  negatively  to  these  items 
than  the  "letter"  groups.  Reactance  theory  may  account  for  the  negative 
reaction  on  these  variables.  However,  reactance  did  not  appear  to  operate 
in  relation  to  the  type  or  number  of  mailings. 

There  appears  to  be  little  research  on  the  "letter"  vs.  "package" 
factor  and  how  it  affects  either  return  rate  or  response  bias.  The 
findings  of  this  study  suggest  that  it  may  be  a worthwhile  factor  to 
explore  in  a variety  of  settings  using  mail  surveys. 

If  differences  among  the  eight  treatment  groups  do,  in  fact,  exist, 
the  finding  of  no  significant  differences  in  return  rate  and  few  consistent 
differences  in  group  means  on  the  12  variables  studied  might  be  due  to 
specific  features  of  the  study  that  failed  to  actually  differentiate 
between  the  two  levels  of  a given  factor.  In  particular,  special  delivery 
letters  in  government-franked  envelopes  may  be  treated  by  recipients  as 
regular  mail  letters  in  such  envelopes.  In  addition,  the  first  and  second 
follow-ups  may  have  been  too  closely  spaced  (1  week  apart).  However,  these 
features  do  not  account  for  significant  differences  between  the  means  of  the 
no  follow-up,  treatment,  and  extensive  follow-up  groups  on  only  2 of  the 
12  variables  investigated. 

This  study  supports  the  assumption  that  the  population  parameters  of 
the  attitudes  Investigated  may  be  more  accurately  estimated,  although  not 
always,  by  obtaining  a larger  sample  of  respondents  through  follow-up 
treatments. 
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To  assess  the  Ronera  1 I stab  l H ty  of  these  results,  liowever,  It  Is  necessary 
to  conduct  further  studies  of  respondents.  Such  studios  should  Include 
variations  of  the  design  characteristics  described  above  and  different 
attitudes  and  populations. 


RECOMMENDATIONS 


1.  Some  kind  of  follow-up  should  be  employed  to  determine  whether  the 
attitudes  measured  produce  a response  bias,  if  the  cost  of  follow-up  is 
low  and  the  risk  of  a small  bias  in  the  estimate  of  the  population  mean  is 
great. 


2.  The  proportion  of  responses  desired,  based  on  the  sample  size, 
should  determine  the  extensiveness  of  follow-ups  used. 

3.  A letter  follow-up  by  regular  mail  is  recommended  over  a duplicate 
package  of  materials  by  special  delivery,  due  to  the  lower  cost  with  no 
consistent  bias  produced. 
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